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Abstract 

A natural generalization of single pattern avoidance is subset avoidance. A com- 



plete study of subset avoidance for the case k = 3 is carried out in SS ] . For k > 3 
situation becomes more complicated, as the number of possible cases grows rapidly. 
Recently, several authors have considered the case of general k when T has some 



nice algebraic properties. Barcucci, Del Lungo, Pergola, and Pinzani in [ BDPP| 



treated the case when T = Ti is the centralizer of k — 1 and k under the natural 



action of Sk on [k]. Mansour and Vainshtein in [MV| treated the case when T = T 2 
is maximal parabolic group of Sk- 

Recently, Babson and Steingrimsson (see [ [BS|] ) introduced generalized permu- 
tations patterns that allow the requirement that two adjacent letters in a pattern 
must be adjacent in the permutation. 

In this paper we present an analogue with generalization for the case T\ and for 
the case T 2 by using generalized patterns instead of classical patterns. 

1. Introduction 

Classical patterns. Let [p] = {1, . . . ,p} denote a totally ordered alphabet on 
p letters, and let a = (ai,...,a m ) G [pi] m , (3 = , • ■ ■ , P m ) £ [P2] m - We say 
that a is order-is omorphic to (3 if for all 1 i < j m one has on < ctj if and 
only if (3i < (3j. For two permutations n 6 S n and r £ Sk, an occurrence of r in 
7r is a subsequence 1 ^ i\ < i 2 < ■ ■ ■ < ik ^ n such that (71^, ...,in k ) is order- 
isomorphic to r; in such a context r is usually called the pattern. We say that it 
avoids t, or is r- avoiding, if there is no occurrence of r in tt. Pattern avoidance 
proved to be a useful language in a variety of seemingly unrelated problems, from 



stack sorting [Kb, Ch. 2.2.1] to singularities of Schubert varieties flLSJ. A natural 
generalization of single pattern avoidance is subset avoidance; that is, we say that 
7T G S n avoids a subset T C Sk if 7r avoids any r e T. A complete study of subset 



avoidance for the case k — 3 is carried out in [3S|. For fc > 3 situation becomes 
more complicated, as the number of possible cases grows rapidly. Recently, several 
authors have considered the case of general fc when T has some nice algebraic 
properties. 



Adin and Roichman in [AR treated the case when T is a Kazhdan-Lusztig 



cell of Sk, or, equivalently, the Knuth equivalence class (see |St| vol. 2, Ch. Al]). 
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Barcucci, Del Lungo, Pergola, and Pinzani in [BDPP] treated the case when T is 
the centralizer of k — 1 and k under the natural action of Sk on [k] . 

Theorem 1.1. (Barcucci, Del Lungo, Pergola, and Pinzani JBDPP| ) Let T be the 

centralizer of k — 1 and k under the natural action of Sk on [k] . Then the ordinary 
generating function for the number of T -avoiding permutations in S n is given by 

(k - ^-ik-l) X -^-^-l )x+{ k-, ?xl + g . [xi 



i=0 



In 2000, Kremer [Kr] presented a generalization for the above theorem, but later 



Mansour [M2, Section 2.3.3] found a counterexample to the main result in Kr 



Recall that the Laguerre polynomial L" (x) is given by 

1 d n 
K{x) = -fx-— (e~V+«) , 

and the rook polynomial of the rectangular sx( board is given by 

R.,t(x)=8lx'L t -'(-x- 1 ) 

for s < t and by R Bt (x) = Rt, s (x) otherwise (see || Ch. 7.4]). In 2000, Mansour 
and Vainshtein [MV] treated the case when we T is a maximal parabolic subgroup 
of S k . 



Theorem 1.2. (Mansour and Vainshtein | MV |) Let 1 < a < m+X, A = min{l,m}, 
H = max{(, m}, and let P'i, m = {& £ Si+ m \a < Oj < a + l — 1, j — 1,2, ... ,1} , then 

A-l r ( l )( m ) At-A-l 

Ft !m {x)R ltm {-x) =J2x r r\J2(- 1 ) jk ^T L + (-1) A ^ A A! £ 
or, equivalently, 



x r r\ 



r=0 



H — r — 1 
A 



k-l 

F l %(x) = '£x r r\ 



l \ (m\ 



j=r+l \ j 

where k = l + m = A + /i 7 .F" m (a;) is tte ordinary generating function for the number 
of P' 'i, m - avoiding permutations in S n - 



Generalized patterns. In [BS] Babson and Steingn'msson introduced gener- 
alized permutation patterns that allow the requirement that two adjacent letters 
in a pattern must be adjacent in the permutation. For example, (generalized) pat- 
terns are 123-4 and 12-34. An occurrence of 123-4 in a permutation tt is a subword 
7rj7r i+1 7rj + 2 7r j °f tt such that 7Tj < 7Tj +1 < 7Tj +2 < Tij! and an occurrence of 12-34 is 
a subword 7Ti7rj+i7r^7r 7 -+i of 7r such that TTj < 7Ti + i < 7Tj < TT-j+i. 

Claesson in presented a complete solution for the number of permutations 
avoiding any single (generalized) pattern of length three with exactly one adjacent 
pair of letters (we do not regard a dash as being a letter) as follows. 



Proposition 1.3. (Claesson Q) For all n > 

, 1, _ f B n if o~ £z {1-23, 3-21, 12-3, 32-1, 1-32, 3-12, 21-3, 23-1}, 
Pn^l-j Cn if a e {2-13,2-31,13-2,31-2}, 

where B n and C n are the nth Bell and Catalan numbers, respectively. 
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In addition, Claesson in Q gave certain results for the number of permutations 
avoiding a pair generalized patterns of three letters. 

Proposition 1.4. (Claesson Q) For all n > 

5„(l-23, 12-3) = B*, S n (l-23, 1-32) = /„, and 5„(l-23, 13-2) = M n , 

where B* is the nth Bessel number (# non- overlapping partitions of [n] (see |F|)); 
/„ is the number of involutions in S n , and M n is the nth Motzkin number. 



Later, Claesson and Mansour [CM1] gave the complete answer for the number 
permutations avoiding a pair patterns of the form ab-c or a-bc where abc G S3. 



In this paper we present an analogue for Theorem 1.1 and for Theorem \L2 by 



using generalized patterns instead of classical patterns as following. 

Let 1 < a < a + l < k and let us denote by C k ; the set of all generalized patterns 
o'ia'2-0'3- . . . -Ok such that (ui, . . . , o^) G Sk, o~i = a, and o~i = a + l. Clearly, 
the set [ contains (k — 2)! generalized patterns (C* t is an analogue for the case 
when T is the centralizer of a and a + I under the natural action of Sk on [k]; 
that is, T = {tt G S^tti = a, tt 2 = a + /}). For example, C\ 2 = {13-2-4, 13-4-2}. 
In the present paper we find the number of ; -avoiding permutations in S n (see 
Section 0) . 

Let be all the generalized patterns of the form o\ . . . ai-ai + \- . . . -a^ such 
that (cti, . . . ,oi) is a permutation of the numbers a, . . . , a + l — 1 and (<7;+i, . . . , (Tfe) 
is a permutation of the numbers a + I, ■ ■ ■ , k, 1, 2, . . . , a — 1. Clearly, P£ l contains 
l\ ■ (k — l)\ generalized patterns (P11 is an analogue for the case when T is a maximal 
parabolic subgroup of Sk] that is, T = {tt G Sk\(^i, ■ ■ ■ , ^1) G Si}). For example, 
P22 contains 4 generalized patterns which are 23-1-4, 23-4-1, 32-1-4, and 32-4-1. 
In the present paper we find the number of P^ r avoiding permutations in S n (see 
Section |). 



2. C% r Avoiding 

Let c k a ;(n) be the number of C^ r avoiding permutations in S n . Our present aim 
is to count this number, since that we introduce another notation. We denote by 
c™ ii,Z2i • • • the number of C^-avoiding permutations tt G S n such that 
t\\t\2 . - - TTj = i\i2 . . .ij. Now we introduce the two quantities that play the crucial 
role in the proof of the main theorem in this section. 

Lemma 2.1. For all n > k, 

n — k+a 

c^(n) = (fe-l)c^(n-l)+ ]T cUn;j). 

3=0- 

Proof. Let it G S n where tt\ = i, and let n > j > n — k + a + 1 or 1 < j < a — 1. By 
definitions it is easy to see that every occurrence of r G C k ; in tt not contains Tt\ , so 

c a,i{ n \j) = c aA n - !)• 0n thc othcr hand - b y definitions c^(n) = YT j= i Ca,i(n;j), 
hence the lemma holds. □ 
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Lemma 2.2. Let n > k, and a<j<n~k + a. Then 

3+1-2 

</(«;i) = (fc- -2)+ £ <,(»-!;«)• 

i— a 

Proof. Let a < j < n — k + a; by definitions 

j— l n 
= £ i>*) + £ C a,/("^''*)' (*) 

i— 1 2=7+1 

and let us consider the possible values of c k i(n;j, i): 

(i) Let 1 < i < a — 1; since all the generalized patterns in C k l started by the 
segment a(a + I) we get 

Ca,i(n;j,i) = c*,(n-2); 

(ii) Let a < i < j — 1; similarly as (i) with a < a + Z we have 

c k a ,i{n;j,i) = c k a<l (n-l;i); 

(Hi) Let j + 1 < i < J + 1 — 1; similarly as (i) we obtain 

4,i( n ;i>*) = c S,i(n-i;*- 1 ); 

(iv) Let J + / < i <n — fc + a + Z; so there exist a%, . . . , afc_2 positions such that 
(j,i,ir ai , . . . ,Tv ah _ 2 ) is order-isomorphic torECj,; hence 

Ca,i(n;j,i) =0; 

(w) Let ri — fc + a + Z + 1 < j < n; similarly as (i) with a < a + I we get 

<l(»5i,*) = c k a ,i(n- 2). 

Hence, if summing c k a i(n;j,i) over all « with using (*) we get the desired result. 

□ 

Now we ready to obtain the main quantity that plays the crucial role in the proof 
of the main theorem in this section. 

Proposition 2.3. Let n > k, and I < i < n — fc+1. Then 
4 !l (n;n-k + a + l-i)= £ (-l)'f (J )(l ^Wn-l-j). 



3=0 



.) 



Proof. Using Lemma 2.1 for i = 1 we get 

n-fc+6-2 

c*,(n;n-fe + a) = (*-l-l)c*,(n-2)+ £ c k a>l {n - l;j). 



Using Lemma 2.2 with ; (n — 1 ; j ) = c k , (n — 2) for 1 < j < a— 1 or ?i — k + a + 1 < 
j < n (see Lemma 2.1), and with a + I > a we have 

c£,j(«;n-* + a) = {k-l-l)c k aJ (n-2) + c k J (n-l)-(a-l + k-a-l + l)c k J (n-2), 
equivalently 

c k j(n; n-k + a) = c k j(n - 1) - c* ,(n - 2). 
Therefore the proposition holds for i = 1. 
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Now, let i > 2; using Lemma 2.1 we obtain for all a + 1 < m < n — k + a 
c£,i(«5 m - 1) = c k a l {n- m) - c* ,(n - 1; m + I - 2), 
and by means of induction on i with employing the familiar identity 



t+1 
s + 1 



we get the desired result. 

Now we are ready to prove the main result of this section. 

Theorem 2.4. For all n > k 

c k a An) = (k-l-l)c k al (n-l)+ 

L(n-fc)/iJ+l 

+ e (-lp'r^r^^X^n-i-i). 

3=0 



(**) 

□ 



Proof. Lemma 2.2 and Proposition |2.3| yield for all n>k 

n-k+l L(i-l)/iJ+l 



t=i j=o 



-1)(/-1) 
J 



Again, employing the familiar identity (**) we get the desired result 



c k a ,i(n-l-j). 

□ 



Example 2.5. (see Claesson and Mansour |CM1 | for k = 3) Theorem 2^4 yields 
for 1 = 1 

n-k+2 

(fc-2)c* il (n-l)= X! (-l) n - fe+2 -^ 



n - fc + 2 

i 



c* il (j + fc-2). 



Using [CM2, Lem. 7] we have 

(k - 2)uC* (1 (u) = (1 + .x) fe - 3 [C7 a fe x - 1 



E 3 (j-i)!^'(fc-2-i(i + w) fc - 3 -^), 

where C^^x) is the ordinary generating function for the sequence {c^ 1 (n)} rl >o, 
hence by putting u — x/(l — x) we get 

C k aA (x) = l + ik^Ml-x)*-^ f^-\+J2(j-l)^(j-(k-2)(l-x) k - 3 -n- 
An infinite number of applications of this identity we have 

^a,l\ x ) ~ 2^ (l-x)(l-2x)...(l-(n-l)x) + 



n>0 



k-3 

E 

j=l n >0 



L^i (1— a)(l-2x)— (l-(n-l)a;) ' 



fc-4 



+ 2^ (l-a;)(l-22;)---(l-na;) ' 



j=0 n>0 



j4n example, for k = 3 we get 

Cl 1 (x)=Cl 1 (x) = J2 



n>0 



(l-x)(l-2x)---(l-(n + l)x) 
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In view of Theorem 2.4 we get the number of C^-avoiding permutations in S n 
is the same number of C\ (-avoiding permutations in S n where a — 1, 2, . . . , k — I. 
Our proof in Theorem 2.4 is an analytical proof. It is a challenge to find a bijective 
proof for this property. 

To put Theorem 2.4 in the form of ordinary generating function let us define as 
follows. Let Z > 2, we define for all n > 0, 



F l n {x) 



m>0 



(-iy 



n-(l- 2)j 

j 



For example, F%(x) = (1 — x) n . 

From now we assume that I > 2. If multiplying the recurrence in the statement 
of Theorem 2.4 by x n and summing over all n > k, then 

n+l 

(k-l-1) £< z (n-l)z™ = £ E(-^ j { n - k+2 -^ m - 1} Ki(n-l-j)x-, 

n>k n>k j—0 

equivalently, 

Theorem 2.6. Let I > 2, then 
x(k-l-l) £ c * I (n)a;" = (* ; -l)!(i^_ 1 ( a: )-l) a ;*- 1 + 

n>fc-l 

+ (fc - 2)\{F l 2l _ 2 {x) - 1 + (21 - 2)x)x k ^ + £ cJ/njF^a, 



For example (see Claesson [Q), for Z = 2 and fc = 3 Theorem 2.6 yields c\ 2 (n) is 
given by the nth Catalan number; that is, the number of permutations in S n (13-2) 

is 4t( 2 ™)- 

n+l V n J 



3. P^- Avoiding 

Let p k j(n) be the number P^j-avoiding permutations in SVi- Our present aim 
is to count this number, since that we introduce another notation. We denote 
by p\ j(n; i\, . . . , i m ) the number P^-avoiding permutations it £ S n such that 
7Ti . . . 7r m = ix . . . i m . Now we introduce the quantity that play the crucial role 
to find p k 

Lemma 3.1. Let a < ij < n — k + I + a — 1 for any j = 1, 2, . . . , m. 

(i) If < m < I — 1, then p k a ; (n; ii, .. . , i m , j) = p k i(n — m — 1) where either 
1 < J < a — 1 ; orn — k + l + a<j<n; 

(ii) J/m = ^en p*j(n;ii, . . . = 0; 

(iii) Por oZ/ i, j(n; t, ...,«,...) = 0. 

Proof, (iii) yields immediately by definitions. To verify (i) let tt 6 <?„ a permuta- 
tions such that TTj — ij for j = 1,2, ... ,m, and 7r m+ i = j. Let r 6 P k , and let 
(7r ai , . . . ,7r af .) order- isomorphic to r. Since m < I — 1, and either j < a - 1 or 
j > n — k + I + a we get ai > m + 1. Therefore, j(n; ii, . . . , i m ,j) is equal to the 
number of P^ ; -avoiding permutations on the letters 1,2, ... ,n without the letters 
ii, . . . ,i m ,j. Hence (i) holds. 
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To verify (ii) let ir £ S n a permutation such that nj — ij for all j = 1,2,...,/, 
so there exist a\ , . . . , ak~i positions such that (ir^ , . . . , ir^ , 7r ai , . . . , n ak _i ) is order- 
isomorphic to r where r <E P^i- Hence pjj ,(n; ii, . . . , i;) = 0. □ 

An application of the above lemma we find the exponential generating function 
for the sequence {p k ; (n)}„>o as follows. 

Theorem 3.2. Let k,l,a>l and 1 < a + I - 1 < k. Then 

(k-i-i)-times 
Proof. By definitions we have 

n 

Pa,lH = ^Pa,/(";*l)- 

ii = l 



Using Lemma 3. 1 (i) for m = we get 

n — fc+Z+a— 1 

p£ ) i(n) = (fc-f)p£,i(»- 1 )+ E pS,i(n;*i)- 

i\ —a 

Now, let us assume for m > 2 

m-l ✓ _ j. , ,\ n-fc+i+a-1 
j=0 »i,...,-i m =l 

so by definitions we have 

m— 1 

<*W-(fc-0 E i!rfHi("-i-i) = 

n — fc+Z+a— 1 n 

= E E Pa,i( n ;*l> ■ ■ ■ )»miWl)) 

ii ,. . .,i m — 1 irM4- i — 1 

equivalent ly 

rn — 1 

^(n)-(fc-O £ j!( n " fe+ !-i) = 

n — k-\-l-\-a— 1 a— 1 n — fc+Z+a— 1 

= E Pa,l(n;«lr'-,»m,Wl)+ E E Pa,i(™; *1> ■ ■ ■ . *m, j) + 

ix f « ■ ■5*771 + 1—1 j — 1 ll , - ■ .,2771=1 

n n — fc+Z+a— 1 

+ E E Pa,i(n;ii,...,i m ,j). 

j=n— k-\-l-\-a ii ,...,^777=1 



Using Lemma 3.1(i)-(iii) we get 

m — 1 

p5»-(*-0 E i!rfRi(«-i-i) = 

3=0 
n — fc+Z+a— 1 

E Pa,i(«;il,---^m,Wl) + (fc-O m! ("~m + 0Pa,i( n - 1 ~ TO )- 

ii,...,i m + l = l 

Hence, by the principle of induction with m = I with Lemma [O] we get for all 

n > 1 

p£,,(») = (* - E J! ( n " * + l )p h aAn 1 - J)- 



3=0 



J 
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Let f(x) = J2n>oPa i(n)x n /nl; if multiplying by x n / (n — k + 1)1 and summing over 
all n > 1 we get 

jfc-; jfc-i-i 

/(a; )=(fc-o5:^-^/(x), 

hence 

A fM - p (k-l)(x/l+x 2 /2+-+x l /l) 

as required. □ 



Again, in view of Theorem 3.2 it is a challenge to find a bijective proof for the 
number of P^-avoiding permutations in S n is the same number of P^ r avoiding 
permutations in S n , for all 1 < a < fc — I +1. 



Example 3.3. (see Mansour flVllfl ) Theorem [5Mi| for I = 1 yields for n> k 

p k a<1 (n) = (k-l)p k aA (n-l), 
with p k Jk) =k\— (k- 1)!, hence p* x (n) = (k - 2)!(fc - l)»- fe + 2 /or alln>k-l. 



Example 3.4. (see Claesson and Mansour [CM1] for k 
I — k — 1 yields for n > k 



3) Theorem 3.& for 



n>0 



In particular, for k — 3 yields the exponential generating function for the number 
of permutations in S n (12-3, 21-3) (or in S n (23-1, 32-1)) is given by e x+x / 2 . 



Again, as a remark, according to the main results (Theorem 2.4 and Theo- 
rem 3.2), C^^x) and P^ z does not depend on a; in other words, |Sn(Cij)| = 
\Sn(C k ,)| and \S n (P k i)\ = \S n (Pai)\ f° r an Y a - We obtained this fact as a conse- 
quence of lengthy computations. A natural question would be to find a bijection 
between S n (C k t ) (respectively, S n (P k l )) and ^(C^,) (respectively, S n (P k l )) that 
explains this phenomenon. 
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